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METHOD AND SYSTEM FOR OBJECT 
RECOGNITION SEARCH 

CROSS-REFERENCE TO RELATED 

APPLICATIONS 5 

This application is related to and claims the benefit of the 
following copending and commonly assigned U.S. Provi- 
sional Patent Application: U.S. Patent Application No. 
61/186,304, titled “Shape and Color Features for Object Rec- 10 
ognition Search,” filed on Jun. 1 1 , 2009; the entire contents of 
which is incorporated herein by reference. 

STATEMENT REGARDING FEDERALLY 
SPONSORED RESEARCH OR DEVELOPMENT ' 

The invention described herein was made in the perfor- 
mance of work under a NASA contract, and is subject to the 
provisions of Public Law 96-517 (35 USC 202) in which the 2 o 
Contractor has elected to retain title. 

BACKGROUND 

1. Field 25 

This disclosure relates to object recognition search. More 

particularly, the present disclosure describes feature extrac- 
tion techniques that are used to facilitate object recognition 
and search. The present disclosure also describes architec- 
tures and systems that are particularly adapted for the recog- 30 
nition and search of moving objects. 

2. Description of Related Art 

While humans are well-adapted for object recognition, 
machine or computer-based object recognition is a challeng- 
ing problem. Several different approaches have been the sub- 35 
ject of much investigation. These approaches include: bio- 
related methods, shape and color features based approaches, 
and image retrieval based shape and color feature methods. 
Bio-related methods have been described by Thomas Serre, 
Lior Wolf, Stanley Bileschi, Maximilian Riesenhuber, and 40 
Tomaso Poggio, in “Robust Object Recognition with Cortex- 
Like Mechanisms”, IEEE Transactions On Pattern Analysis 
And Machine Intelligence, Vol. 29, No. 3, 411-426, March 
2007; by Fukushima, K., in “Neocognitron: a self-organizing 
neural network model for a mechanism of pattern recognition 45 
unaffected by shift in position,” Biol. Cybem., 36: 193-202 
(1980); by Riesenhuber, M. and Poggio, T., in “Hierarchical 
models of object recognition in cortex”. Nat. Neurosci., 2: 
1019-1025 (1999); and Perrett, D. I. and Oram, M., in “Neu- 
rophysiology of shape processing,” Image Vis. Comput., 11: 50 
317-333 (1993). Shape and color features based approaches 
have been described by B. W. Mel, in “SEEMORE: Combin- 
ing Color, Shape, and Texture Histogramming in a Neurally 
Inspired Approach to Visual Object recognition,” Neural 
Computation, 9, pp. 777-804, 1997 and by Eitan Sharon, 55 
Meirav Galun, Dahlia Sharon, Ronen Basri & Achi Brandt, in 
Nature, Vol 442.17, 810-813, August 2006. Image retrieval 
based shape and color feature methods have been described 
by Theo Gevers and Arnold W. M. Smeulders, in“PicToSeek: 
Combining Color and Shape Invariant Features for Image 60 
Retrieval,” IEEE Transactions On Image Processing, Vol. 9, 
No. 1, pp. 102-119, January 2000, and by A. K. Jain and A. 
Vailaya, in “Image retrieval using color and shape” Pattern 
Recognition, vol. 29, pp. 1233-1244, 1996. Since the shape 
and color features of a given object can uniquely define its 65 
characteristics and they also agree with biological models, 
these approaches are strongly related. 


2 

From a physical perspective, an electronic image of an 
object is a collection of photon reflections from the surface of 
that object with respect to a fixed camera position. If the 
object moves and the motion/rotation of the surface of the 
object are sufficiently small, a new collection of photon 
reflections may not be correlated with the previous reflections 
from a pixel-wise view point; however, they still maintain a 
strong correlation with a global view of the object. To obtain 
a global view of an object, the shape feature is the logical 
building block to be used and the color feature, if available, 
provides additional information. Moreover, for different 
lighting conditions and different light absorption properties 
of the various materials in the object, it reflects and absorbs 
differently, so that the object color provides a unique 
response. When the object moves in an evolving light envi- 
ronment, the color response will change locally with respect 
to its previous response. 

Shape Feature Extraction 

For problems with rich data sources such as image recog- 
nition and computer vision, the dimension of the input vector 
is typically larger than the number of input samples, which 
leads to overfitting of the data if no care is taken to achieve 
useful generalization. Furthermore, computing power rapidly 
becomes insufficient to process the data within a reasonable 
time. To overcome these two obstacles, a preprocessing step 
may be effective in the following two ways: (1) non-useful 
and redundant data are eliminated, thus enhancing the opera- 
tional processing step; and, (2) the salient feature can be 
selected, thus improving the recognition capability. 

Support vector machines, learning by hints, Principal 
Component Analysis (PCA) and Cellular Neural Networks 
(CNN) all reduce the dimension of the input vector set with 
little or no significant loss of information. Such data extrac- 
tion methods also have the advantage of eliminating some 
irrelevant data, such as small amplitude noise, and speeding 
up the classification step. Sequential PCA is a tool that may 
require less computation and be more hardware friendly to 
enable real time feature extraction for real time adaptive 
capability. 

Principal Component Analysis (PCA) is a second order 
statistical approach, which can been used to extract the fea- 
tures of a data set (see, for example, A. K. Jain, R. P. W. Duin, 
and J. Mao “Statistical Pattern Recognition: A Review”, 
IEEE Transactions on Pattern Analysis and Machine Intelli- 
gence, Vol. 22, NO. 1, January 2000) or to perform data 
compression (see, for example, T. A. Duong “Real Time 
On-Chip Sequential Adaptive Principal Component Analysis 
for Data Feature Extraction and Image Compression”, 
GOMAC Tech-03, Vol. I, pp. 460-464, Tampa, Fla, 31 
March-3 April, 2003; T. A. Duong andV. A. Duong, “Sequen- 
tial Principal Component Analysis — An Optimal and Hard- 
ware-Implementable Transform for Image Compression”, 
3rd IEEE Space Mission Challenges for Information Tech- 
nology in Pasadena, Calif, 19-23 July, 2009; and S. Bannour 
and M. R. Azimi-Sadjadi, “Principal Component Extraction 
Using Recursive Least Squares Learning,” IEEE Trans. On 
Neural Networks, Vol. 6, No. 2, March 1995). Especially, 
when the data set is Gaussian, redundant and overwhelmingly 
large, PCA is a very effective preprocessing step to extract 
data features for classification and/or to cluster data in the 
most compact energy vectors for data compression. Unfortu- 
nately, PCA requires that the basis vectors be orthogonal, 
which is typically an artificial assumption. 

The PCA procedure is complicated and computationally 
intensive (0(N3), where N is the dimension of the vector 
input), thereby making it difficult to use for rich data sources. 
To get over the hurdles from the traditional PCA technique, 
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simple sequential PCA techniques have been developed (see, 
for example, E. Oja and J. Karhunen, “On stochastic approxi- 
mation of the eigenvectors and eigenvalues of the expectation 
of a random matrix.” J. Math. Anal. Appl., vol. 106, pp. 69-84, 
1985). These techniques are based on a learning approach that 5 
sequentially obtains principal component vectors. Some 
works in PCA are reported using Hebbian or anti-Hebbian 
learning (see, for example, S. Haykin, Neural Network — A 
Comprehensive Foundation. New York: Macmillan, 1994, 
and P. Baldi and K. Hornik, “Learning in linear neural net- to 
works: A survey,” IEEE Trans. Neural Networks, Vol. 6, pp. 
837-857, 1995) and gradient-based learning (see, for 
example, S. Bannour and M. R. Azimi-Sadjadi, “Principal 
Component Extraction Using Recursive Least Squares 
Learning,” IEEE Trans. On Neural Networks, Vol. 6, No. 2, 15 
March 1995., and L. Xu, “Least mean square error recon- 
struction principal for self-organizing neural-nets,” Neural 
Networks, 6, pp. 627-648, 1993) or even the more elegant 
technique of natural gradient descent (see, for example, S. I. 
Amari . , “Natural gradient works efficiently in learning,” Neu- 20 
ral Computation, 1998). 

For sequential PCA, the gradient descent (GED) technique 
may be a more attractive approach for hardware implemen- 
tation as a straight forward technique compared to others, 
e.g., steepest decent, conjugate gradient, or Newton’s second 25 
order method, but it exposes some difficulties in learning 
convergence when other principal component vectors are cor- 
responding to smaller eigenvalues. In addition, this technique 
still requires some complicated hardware. 

A Dominant-Element-Based Gradient Descent and 3Q 
Dynamic Initial Learning Rate technique for sequential PCA 
has been developed. This technique serves two purposes: 1) 
simplified hardware implementation, especially in VLSI as 
System-On- A-Chip approach; and 2) fast and reliable con- 
vergence as compared with counterpart gradient descent. 
This technique requires much less computation and its opti- 35 
mized architecture is more suitable for implementing as a real 
time adaptive learning system in hardware. 

The objective function for the Dominant-Element-Based 
Gradient Descent and Dynamic Initial Learning Rate tech- 
nique is defined as shown in Eq. 1 below: 40 

m m k Eq. 1 

J (w) = 2 = 2 Xj ” W i W J X t I 2 

i=l i=l r=l 

where m is the number of principal components, k is the 45 
number of measurement vectors, x„ measured at time t and w z 
is the \ th principal vector (or eigen vector). 

From Eq. 1 , the additional definitions shown in Eq. 2 and 
Eq. 3 below can be made: 


k 

Ji(wi) = ^ j \y t i-wiw[y t i\ 2 

t = 1 

Eq. 2 

/- 1 

y f i = x t -J]wjWjX t 

Eq. 3 

55 


J= 1 


From Eq. 2 and Eq. 3, the learning algorithm can be pro- 
cessed sequentially for each principal vector that is based on 
the gradient descent as shown in Eq. 4 below: 


dJi d(\$-w i w[y t i \ 2 ) E< 1- 4 


65 

From Eq. 4, only the dominant element (see T. A. Duong 
“Real Time On-Chip Sequential Adaptive Principal Compo- 


4 

nent Analysis for Data Feature Extraction and Image Com- 
pression”, GOMAC Tech-03, Vol. I, pp. 460-464, Tampa, 
Fla., 31 Mar. -3 Apr., 2003) is used; the weight update can be 
obtained as shown in Eq. 5 below: 


+ f M + »Wy) Eq - 5 

where £ = — — and y\ = w;wf y]. 


E 0 is the initial energy when the network starts learning and 
E z-1 is the energy of the (\-\) th extracted principal. 

The techniques described above may be used in the extrac- 
tion of shape features for object recognition. Application of 
these techniques to embodiments of the present invention is 
described below in the Detailed Description section. 

Color Feature Extraction 

To use the object color feature for helping in object recog- 
nition in a friendly and correlated environment, color seg- 
mentation is a suitable approach to narrow down the search 
space. Several color segmentation algorithms have been pro- 
posed in literature (see, for example, M. Celenk. “A Color 
Clustering Technique for Image Segmentation.” Computer 
Vision Graphics Image Process. Graphical Models Image 
Process. 52. pp. 145-170, 1990; J. Lui andY. H. Yang, “Mul- 
tiresolution Color Image Segmentation,” IEEE Trans. Patt. 
Anal. Mach. Intel. 16, 689-700, 1994; E. Littman and H. 
Ritter. “Adaptive Color Segmentation — A Comparison of 
Neural and Statistical Methods.” IEEE Trans. Neural Net. 
Vol. 8, No. 1, pp. 175-185, 1997; F. Perez and C. Kock, 
“Toward Color Image Segmentation in Analog VLSI: Algo- 
rithm and Hardware,” Int. J. Comp. Vision 12:1, 17-24, 1 994; 
G. Healy. “Segmenting Images Using Normalized Color.” 
IEEE Trans. Syst. Man. Cyber. 22. 1. pp. 64-73, 1992; H. 
Okii, et al. “Automatic color segmentation method using a 
neural network model for stained images,” IEICE Trans. Inf. 
Syst. (Japan) Vol. E770DNo. 3, pp. 343-350, March 1994; T. 
Nakamura and T. Ogasawara, “On-Line Visual Learning 
Method for Color Image Segmentation and Object Tracking,” 
Proc. of The 1999 IEEE/RSJ Intelligent Robots and Systems, 
pp. 222-228, 1999; P. H. Batavia and S. Singh, “Obstacle 
Detection Using Adaptive Color Segmentation and Color 
Stereo Homography,” Proc. Of the 2001 IEEE International 
Conference on Robotic and Automation, Seoul, Korea, May 
21-26, 2001; E. Fiesler, S. Campbell, L. Kempen, and T. 
Duong. “Color Sensor and Neural Processor on One Chip.” 
International Symposium on Optical Science, Engineering, 
and Instrumentation, Proc. of the SPIE, vol. 3455 ‘Applica- 
tions and Science of Neural Networks, Fuzzy Systems, and 
Evolutionary Computation’, pp. 214-221 , 1 998; T. A. Duong, 
“Real Time Adaptive Color Segmentation for Mars Landing 
Site Identification,” Journal of Advanced Computational 
Intelligence and Intelligent Informatics (Japan), pp. 289-293, 
Vol. 7 No. 3, 2003). 

The majority of existing color segmentation techniques are 
based on Red-Green-Blue (RGB) classification in combina- 
tion with complex data processing. This processing includes 
spatial clustering to separate targets from the background, 
multi-histogram analysis (see, for example, M. Celenk. “A 
Color Clustering Technique for Image Segmentation.” Com- 
puter Vision Graphics Image Process. Graphical Models 
Image Process. 52. pp. 145-170, 1990), Bayesian methods 
(see, for example, J. Lui and Y. H. Yang, “Multiresolution 
Color Image Segmentation,” IEEE Trans. Patt. Anal. Mach. 
Intel. 16, 689-700, 1994), and various neural network 
approaches (see, for example, E. Littman and H. Ritter. 
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“Adaptive Color Segmentation — A Comparison of Neural 
and Statistical Methods.” IEEE Trans. Neural Net. Vol. 8 , No. 
l,pp. 175-185, 1997). Algorithms have been proposed based 
on edge detection in color space (see, for example, F. Perez 
and C. Kock, “Toward Color Image Segmentation in Analog 5 
VLSI: Algorithm and Hardware,” Int. J. Comp. Vision 12:1, 
17-24, 1994) and normalized color space (see, for example, 

H. Okii, et al. “Automatic color segmentation method using a 
neural network model for stained images,” IEICE Trans. Inf. 
Syst. (Japan) Vol. E770D No. 3, pp. 343-350, March 1994). 10 

In real time applications, algorithms with fast learning and 
adaptive capabilities are preferred. As described in T. A. 
Duong and Allen R. Stubberud, “Convergence Analysis Of 
Cascade Error Proj ection — An Efficient Learning Algorithm 1 5 
For Hardware Implementation”, International Journal of 
Neural System, Vol. 10, No. 3, pp. 199-210, June 2000, Cas- 
cade Error Projection (CEP) algorithm provides an excellent 
tool in fast and simple learning. 

The CEP neural network architecture is illustrated in FIG. 2 o 

I . Shaded squares 102 and circles 103 indicate frozen 
weights; a non-shaded square 112 indicates calculated 
weights, and a non-shaded circle 113 indicates learned 
weights. As can be seen in FIG. 1 , the shaded squares 102 and 
shaded circles 103 indicate the learned or calculated weight 2 5 
set that is computed and frozen. A non-shaded circle 113 
indicates that perceptron learning is applied to obtain the 
weight set and a non-shaded square 112 indicates that the 
weight set is deterministically calculated. 

In the CEP algorithm, the energy function is defined as 30 
shown in Eq. 6 below: 


E{n + 1) = 


P ( m \2 


Eq. 6 
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The weight update between the inputs (including previ- 
ously added hidden units) and the newly added hidden unit is 
calculated as shown in Eq. 7 below: 


A>4(/1+ 1) = - 7 ] 


dE(n + 1) 
dwf h (n + 1) 


Eq. 7 
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and the weight update between hidden unit n + 1 and the output 
unit o is as shown in Eq. 8 below: 


1 ) 

w ho (n + 1) = 

Y J Uo P ti(n+ l )] 2 

P= 1 


with f(x) = 


1 - e x 
1 + e~ x ' 
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Eq. 8 
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where m is the number of outputs and P is the number of 
training patterns. Error € 0 p =t 0 p -o 0 p (n); where o/(n) is the 
output element o of the actual output o(n) for training pattern 60 
p, and Xf is the target element o for training pattern p. n 
indicates the number of previously added hidden units, f p 
(n )=f Q p denotes the output transfer function derivative with 
respect to its input, ^(n+l) denotes the transfer function of 
hidden unit n+1. 65 

The CEP algorithm is processed in two steps: (1) Single 
Perceptron learning which is governed by Eq. 7 to update the 


weight vector W z 7 z (n+1); and (2) when the single Perceptron 
learning is completed, the weight set W^ G (n+l) can be 
obtained by the calculation governed by Eq. 8 . Additional 
details of the CEP algorithm and the convergence analysis 
may be found in T. A. Duong and Allen R. Stubberud, “Con- 
vergence Analysis Of Cascade Error Projection — An Effi- 
cient Learning Algorithm For Hardware Implementation”, 
International Journal of Neural System, Vol. 10, No. 3, pp. 
199-210, June 2000. 

While techniques are known in the art for object detection 
and recognition based on either shape feature extraction and 
detection or color feature extraction and detection, the use- 
fulness of these techniques for use in a heterogeneous envi- 
ronment may be somewhat limited. Hence, there is a need in 
the art for techniques that will support effective recognition in 
more widely varying environments. 

SUMMARY 

Embodiments of the present invention may use a bio -in- 
spired features approach, an adaptive architecture and 
autonomous mechanism, and an effective processing tech- 
nique to enable a real time adaptive visual system. From a 
physiological and psychological view point, there is evidence 
that the brain may process a sub-shape feature as a part -based 
shape, e.g., mouth, eyes, etc in the face, for object recogni- 
tion. This is also evident in the saccadic eye movement for a 
part-based object. Shape feature extraction and detection 
according to embodiments of the present invention emulate 
saccadic eye movement by focusing on smaller size and 
smaller samples of a scene where each scene has different 
sample location. 

One embodiment of the present invention is a method for 
object identification in multiple images in a series of images, 
the method comprising: extracting initial knowledge of color 
and shape features from a statistical image collection and 
storing the extracted color and shape features in one or more 
databases; performing shape feature detection and color fea- 
ture detection on a first image in the series of images to 
identify an object and a region of interest within the first 
image, wherein the shape feature detection and color feature 
detection is based on the color and shape features stored in the 
one or more databases; extracting new color and shape fea- 
tures based on the identified region of interest and storing the 
new extracted color and shape features in the one or more 
databases; performing shape feature detection and color fea- 
ture detection on a next image in the series of images to 
identify the object and a region of interest within the next 
image, wherein the shape feature detection and color feature 
detection is based on the color and shape features stored in the 
one or more databases; and repeatedly extracting new color 
and shape features followed by performing shape feature 
detection and color feature identification to identify the obj ect 
in multiple images in the series of images. 

Another embodiment of the present invention is a com- 
puter programmed for recognition of an object in multiple 
images of a series of images comprising: one or more data- 
bases for storing color and shape features extracted from a 
statistical image collection containing images of objects to be 
recognized; one or more processors programmed to extract 
shape features from a region of interest or an obj ect of interest 
within an image and store the extracted shape features in the 
one or more databases; the one or more processors being 
further programmed to extract color features from a region of 
interest or an object of interest within an image and store the 
extracted color features in the one or more databases; the one 
or more processors being further programmed to detect shape 
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features and color features within an image or a region of 
interest based on shape features stored in the one or more 
databases to recognize an object with the image or the region 
of interest; the one or more processors programmed to extract 
shape features and color features being further programmed 5 
to extract shape features and color features from the recog- 
nized object or region of interest, where the one or more 
processors are further programmed to provide an indication 
of the presence of the recognized object in one or more 
images of the series of images. 10 

BRIEF DESCRIPTION OF THE SEVERAL 
VIEWS OF THE DRAWINGS 

FIG. 1 (prior art) illustrates a Cascade Error Projection 15 
neural network architecture. 

FIG. 2 illustrates a processing architecture for recognizing 
bio-inspired features of an image or person in images. 

FIGS. 3A-3C show training images for persons selected 
from an image database. 20 

FIGS. 4A-4C show images in the database for the person 
shown in FIG. 3A. 

FIGS. 5A-5C show images in the database for the person 
shown in FIG. 3B. 

FIGS. 6A-6C show images in the database for the person 25 
shown in FIG. 3C. 

FIG. 7 shows a series of images for the person shown in 
FIG. 3A. 

FIG. 8 shows a moving object sampled from a scene. 

FIG. 9A shows a test image and FIG. 9B shows a detected 30 
object found in the test image based on the object shown in 
FIG. 8. 

FIG. 10A shows another test image and FIG. 10B shows a 
detected object found in the test image based on the object 
shown in FIG. 8. 35 

FIG. 11 A shows another test image and FIG. 11B shows a 
detected object found in the test image based on the object 
shown in FIG. 8. 

FIG. 12A shows another test image and FIG. 12B shows a 
detected object found in the test image based on the object 40 
shown in FIG. 8. 

FIG. 13 A shows another test image and FIG. 13B shows a 
detected object found in the test image based on the object 
shown in FIG. 8. 

FIG. 14 A shows another test image and FIG. 14B shows a 45 
detected object found in the test image based on the object 
shown in FIG. 8. 

FIG. 15A shows an image of an enviromnental scene and 
FIG. 15B shows a segmented representation of the image 
shown in FIG. 15A. 50 

FIG. 16 A shows a later image of the scene shown in FIG. 
15A, FIG. 16B shows a segmented image based on previous 
training data from the image shown in FIG. 15A, and FIG. 

1 6C is a segmented image resulting from intermediate knowl- 
edge through adaptive learning. 55 

FIG. 17A shows a later image of the scene shown in FIG. 
15A, FIG. 17B shows a segmented image based on previous 
training data from the image shown in FIG. 16 A, and FIG. 
17C is a segmented image resulting from intermediate knowl- 
edge through adaptive learning. 60 

FIG. 18 shows a flow chart for object recognition using 
shape and color feature integration. 

FIG. 19 A shows an initial scene and FIG. 19B shows a 
selected object within that scene. 

FIG. 20 shows a video sequence of images containing the 65 
object shown in FIG. 19B. 

FIG. 21 depicts saccadic eye movements. 
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FIG. 22 shows a sample scene and a subject from which 
shape features are extracted. 

FIGS. 23A-23E depict the shape features extracted from 
the subject shown in FIG. 22. 

DETAILED DESCRIPTION 

The exemplary embodiments according to the present 
invention described in this disclosure provide an adaptive 
capability for shape recognition and color recognition based 
on local color and shape correlation for the recognition of a 
moving object in an evolving environment. 

A major challenge for a color and shape feature based 
recognition technique is how to obtain the effective features 
in real time, or near real time, which enable an adaptive 
capability for dynamically changing objects in a 3-D envi- 
ronment. When the operating environment is evolving, the 
change of shape and color features are not guaranteed to be 
local or ‘smooth’; however, embodiments of the present 
invention may rely on the assumption that at least one of them 
changes ‘smoothly’. The feedback architecture of the recog- 
nition system uses two variables, the color and shape feature 
vectors, as input parameters to drive the system to optimally 
converge to the correct attractor i.e., to the correct object 
recognition. 

Embodiments of the present invention may employ a fea- 
ture database that has adapted and stored shape and color 
features. When assumptions of smoothness of shape and 
color features in time are violated, the feature database may 
be employed to search the match features from scratch as a 
new start. 

From a biological view point, the saccadic eye movement 
is well understood; however, its use in visual models has not 
gone beyond recognition of the phenomenon. In some 
embodiments of the present invention, saccadic eye move- 
ments are mapped into a statistical form from which Principal 
Component Analysis (PCA) can be used to extract the fea- 
tures when a single (or multiple) visual sensor object is pro- 
vided. One can view this approach as a bio-inspired model 
which emulates the saccadic eye movement and the horizon- 
tal layer in the retina of a vertebrate. From this modeling, the 
significant and salient shape features of the object of interest 
can be extracted. 

Embodiments of the present invention emulate saccadic 
eye movement by focusing on smaller size and smaller 
samples of a scene where each scene has different sample 
location (spatial location). From different samples based on 
saccadic eye movement e.g., spatial location, one can have a 
set of sample data to enable the statistical technique to extract 
the bio-inspired feature for that object. For example, an upper 
left corner of an image may be used as a reference. Then the 
spatial location of each sample image is sampled to build a 
statistical sample set for shape feature extraction. Preferably, 
a pattern that is randomly based on the light intensity and 
perception of the task will be used to build the statistical 
sample. For example, that pattern may be based on perceiving 
a single subject or a group of subjects. Such perception tasks 
may require different sampling tactics. Saccadic eye move- 
ment was recorded by Yabus, 1950 (see FIG. 21). Embodi- 
ments of the present invention emulate this approach as 
shown in FIG. 21 . 

FIG. 22 shows a sample scene and a window 903 around a 
subject from whom shape features are to be extracted. FIGS. 
23A-23E show five features extracted from the subject within 
the window according to embodiments of the present inven- 
tion. As seen in FIG. 23A-23E, the various features may be 
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predominated by general shape of the head, the positions of 
the eyes relative to the hair, etc. 

As discussed above in regard to Equations 1-5, a Domi- 
nant-Element-Based Gradient Descent and Dynamic Initial 
Learning Rate technique may be used for shape feature 5 
extraction according to some embodiments of the present 
invention. Embodiments of the present invention using such a 
technique may allow for a relatively simple and optimized 
hardware architecture for shape feature extraction. Other 
embodiments of the present invention may use alternative 10 
approaches for shape feature extraction. 

Described below are two experiments that were conducted 
based on a bio-inspired shape feature approach according to 
embodiments of the present invention. In the first experiment, 
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The experiment was based on the use of one sample image, 
since, in real world applications, only a single image may be 
available for performing recognition. Moreover, the search 
results may depend on the users’ desire by controlling the 
threshold value to obtain a pool of candidate matches and its 
size. 

In this experiment, the constraints were not only to maxi- 
mize the percentage of correct recognition, but also maintain 
a good generalization capability. Per each sample image/ 
person, only about 2.4% of the total image of person had to be 
inputted to the system. Based on that input, the average cor- 
rect recognition was about 5 5% of that person’ s appearance in 
the entire database. To obtain that percentage, the threshold 
controlled values ere relaxed, which introduced more false 
positives. Simulation results are shown in Table I. 


TABLE I 

Features FI F1&F2 F1-F3 Comments 


Percentage of correct 86% 92% 

recognition with 
positive false alarm 

Percentage of correct 96.7% 

recognition with no 
positive false alarm 


94.2% With one sample input per person 
(2.4%) and 55% correct 
generalization (20 different images 
of the same person) 

15% correct generalization (7 
different images of the same 
person) with no positive false alarm 


PC A feature extraction was performed for static object rec- 
ognition. In the second experiment, PCA feature extraction 
was performed for dynamic object recognition. 

In the first experiment, an image retrieval application based 
on object recognition is addressed, where the image database 
and searched image are available. For example, a search may 
be performed for a specific person whose appearance is 
known from a Department of Motor Vehicles driver license 
image database. In this application, only a single image is 
obtained and used as an image input to prompt the database 
for the best match. 

For the experiment, a database of 1520 images (286x384 
pixel resolutions) of 23 different persons was used. This data- 
base may be available via the Internet at the website http:// 
www.humanscan.de/support/downloads/facedb.php. In the 
experiment, a single image of one random person at a time 
was used for bio-inspired feature extraction. Three bio-in- 
spired features of an image of each person were used. The 
processing architecture for processing the images is shown in 
FIG. 2 

3 persons of the 23 persons in the database were selected 
and a selected image per person was chosen. The selected 
images are shown in FIGS. 3 A, 3B and 3C. Each image was 
resized to 50% of the original in order to improve processing 
time. These images were processed to obtain three bio-fea- 
tures (eigen vectors) for each selected image. These feature 
vectors were then used to correlate through a test image to 
find the maximum correlation within it. The maximum cor- 
relation values for each of the 1520 images in the database 
where then thresholded to determine whether the image 
belonged to the same person. 

FIGS. 4A-4C show the images in the database for the 
person shown in FIG. 3A. In the experiment, the images 
shown in FIGS. 4A and 4B were correctly selected, while the 
image shown in FIG. 4C was missed. Similarly, FIGS. 5A and 
5B show the images correctly selected based on FIG. 4B, 
while FIG. 5C shows a missed image. Finally, FIGS. 6A and 
6B show the images correctly selected based on FIG. 4C, 
while FIG. 6C shows a missed image. 


As shown in Table I, 96.7% correct recognition with no 
positive false alarm using only one feature (FI) was obtained 
with the relaxation of the generalization to 15% (instead of 
55%) correct recognition of a person appearance in the entire 
database (with 2.4% input image or one image/per person) 
with no positive false alarm using only one feature (FI). For 
example, the first feature (FI) of the person in FIG. 3A was 
used to recognize the same person in all pictures shown in 
FIG. 7. 

The simulation results were excellent due to the simplified 
and practical model. Moreover, the scaling and rotation tol- 
erances are observed in this approach. The time consuming 
from selecting the object (face) to feature extraction and 
search for 1520 images is about 924 seconds. 

The discussion above addresses static object recognition, 
that is, recognition of an object from a series of still pictures. 
However, dynamic object recognition, where an object is 
recognized from a series of images in which the object to be 
recognized is moving through a scene, provides additional 
applications in a real world environment e.g., intelligent 
video search etc. Hence, a practical model for dynamic object 
recognition would be useful. 

In an embodiment according to the present invention, bio- 
inspired shape features of an object are obtained. These fea- 
tures are then used to detect the object in the new scene based 
on their correlations. The correlation value of each shape 
feature corresponding to the similar raw image must be clos- 
est to eigenvalue of each corresponding feature vector (eigen 
vector). After the object is detected, then the newly found 
object is automatically feedback into the extraction processes 
so that new features may be extracted. 

The applicability of this approach was studied by testing 
moving object recognition using grayscale images. Due to the 
use of grayscale images, only the adaptive shape feature 
approach is employed. This study demonstrates the adaptive 
shape feature mechanism and moving object recognition 
approach and shows the applicability of such an approach in 
a shape and color feature mechanism approach according to 
an embodiment of the present invention. 
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A moving car object acquired from the USC-SIPI Image 
Data Base (available at in http://sipi.usc.edu/database/) was 
used as a benchmark. This sequencing image is carefully 
designed to show that the car has moved in the orthogonal 
direction to avoid a trivial movement. FIG. 8 shows a moving 
object (100x100) sampled from a scene. Only one feature 
(eigen vector) was used for the subsequent object recognition 
discussed below. 

FIG. 9A shows a test image and FIG. 9B shows the detected 
object found in the test image that is the same as the initial 
object shown in FIG. 8. Similarly, FIG. 10A shows another 
test image and FIG. 10B shows the detected object found. 
FIG. 11A shows still another test image and FIG. 11B shows 
the detected object. FIG. 12A shows still another test image 
and FIG. 12B shows the detected object. FIG. 13A shows still 
another test image and FIG. 13B shows the detected object. 
Finally, FIG. 14A shows still another test image and FIG. 14B 
shows the detected object. As shown by these figures, the 
bio -inspired shape feature successfully detected the moving 
car based on adaptive shape feature in the dynamic scene. 
However, note that a fixed window size was used in this 
simulation, so the detected objects are not fully exposed in the 
window, as shown in FIGS. 9B, 10B, 11B, 12B, 13B and!4B. 

The embodiments described utilize powerful bio-inspired 
feature extraction, fast PCA processing techniques and adap- 
tive shape feature architecture and autonomous feedback 
mechanism for object detection and recognition. The combi- 
nation of these techniques enables real time adaptive capabil- 
ity, which can be an effective technique to address object 
recognition in a changing environment. 

In a heterogeneous environment, the shape of an object 
may itself expose difficulties to recognition when: a) the 
object change violates the assumption of a “smooth” transi- 
tion from which the adaptive approach may not be sufficient; 
or b) an ambiguous environment exists where several similar 
types of objects exist. To deal with this issue, the color fea- 
tures of an object may be used to extend a new dimension of 
search. Color segmentation may be an optimal technique to 
be used to assist the shape feature for object recognition. 
Color segmentation techniques known in the art are discussed 
above, including those using algorithms with fast learning 
and adaptive capabilities, such as the CEP algorithm. 
Embodiments of the present invention may use the CEP algo- 
rithm and the CEP neural network architecture for object 
color feature detection and extraction. 

A challenging aspect in color segmentation is when the 
light intensity and resolution are dynamically changing. The 
initial knowledge used to train the CEP network will have 
very little effect at a new location and therefore will need to be 
updated through learning of newly extracted data. When the 
network that has acquired current color knowledge at time t 0 
is used to test the subsequent image at time t 0 +At, segmenta- 
tion results from the image at t 0 + At will be used to extract the 
training set to update the previous knowledge of the network 
at time t 0 . This process of repeatedly segmenting and updat- 
ing is performed as long as the operation is going on. 

While the process of segmenting and updating are desired 
characteristics of an adaptive processing, there is an issue as 
to how often such updates are necessary. The problem with 
infrequent updates, however, is that the network may not 
interpolate easily based upon new images from which the 
newly segmented data may be insufficient for training. To find 
the optimal sampling rate, At must be “sufficiently small” and 
will depend upon the moving velocity and other environmen- 
tal changes. 

A simulation of the color feature extraction using images 
having environmental features was performed. To classify 
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each pixel in the images, a pixel to be classified and its 
immediate neighbors were used to form a 3x3 sub- window as 
the input training pattern (thus each input pattern has 27 
elements from 3x3 of RGB pixel). Based on a previous study 
5 (see E. Fiesler, S. Campbell, L. Kempen, and T. Duong. 
“Color Sensor and Neural Processor on One Chip.” Interna- 
tional Symposium on Optical Science, Engineering, and 
Instrumentation, Proc. of the SPIE, vol. 3455 ‘Applications 
and Science of Neural Networks, Fuzzy Systems, and Evo- 
10 lutionary Computation’, pp. 214-221, 1998), the 3x3 RGB 
input pattern was found to be the optimal size in comparison 
to using a single pixel RGB input, a 5x5 pixel RGB sub- 
window, or a 7x7 pixel RGB sub-window. In this simulation, 
the objective was to segment the image into three groups: 
15 “Rockl”, “Rock2”, and “Sand”. The topology of the CEP 
network in the simulation was a 27x5x3 cascading architec- 
ture neural network, having 27 inputs, 5 cascaded hidden 
units, and 3 output units. 

FIG. 15A shows an image of an environmental scene at 3 
20 PM. FIG. 15 A shows “Rockl” 701, “Rock2” 703, and 
“Sand” 705. This image was sampled and 408 patterns were 
collected for training data, 588 patterns were collected for 
cross validation, and 1200 patterns were collected for train- 
ing. With these sample sets, the learning was completed with 
25 91% correct in training, 90% correct in validation, and 91% 
correct in testing. After training was performed, FIG. 15B 
shows the segmented output of the original image shown in 
FIG. 15A. 

With the knowledge acquired in FIG. 15A, thenetworkwas 
30 tested with the image input shown in FIG. 16A that was 
collected at 4:00 PM (resulting in both a positional change of 
the objects of interest and a change in the shadows associated 
with the objects). The output result is shown in FIG. 16B 
(where the segmented image is based on training data from 
35 the image shown in FIG. 15 A where no intermediate adapta- 
tion step was performed). FIG. 16C is an output result with 
the network acquired from the intermediate knowledge 
through adaptive learning. In a similar manner, the original 
image shown in FIG. 17A was collected at 5 PM. FIG. 17B is 
40 the segmented image with the previous training set at 4 PM 
and FIG. 17C is the segmented image with intermediate adap- 
tive step. 

Based on the aforementioned results, it may be concluded 
that the adaptive technique is needed to obtain better segmen- 
45 tation when the environment is changing rapidly. 

As discussed earlier, the role of shape feature itself carries 
substantial information for recognition while the color fea- 
ture may help to ease the unexpected event which may often 
occur in the heterogeneous environment. Embodiments of the 
50 present invention utilize an integration of the shape and color 
feature based approaches. Such an integration of approaches 
helps strengthen object recognition to meet object recogni- 
tion challenges of real world applications. 

FIG. 18 shows a flow chart of an embodiment according to 
55 the present invention having shape and color feature integra- 
tion. As shown in FIG. 18, feature extraction 806 is performed 
on a statistical image collection (either single or multiple) to 
create an initial knowledge of shape feature(s) that is stored in 
an adaptive shape feature database 802. Similarly, color 
60 detection 807 is performed on a statistical image collection 
(either single or multiple) to create an initial knowledge of 
color feature(s) that is stored in an adaptive color feature 
database 807. At time t, an object or region of interest 805 is 
tested and found based on an initial shape and color feature. 
65 Then, feature extraction 805 is performed on the object or 
region of interest 805 to get a new shape feature or features 
and color detection 807 (preferably via segmentation) is per- 
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formed to get a new color feature or features at time t+At. If 
there is a significant difference between the previous and new 
features, the color or shape features will be updated and 
stored in the feature database 

From the previous shape adaptive features and color adap- 
tive features stored in the shape feature database 802 and 
color feature database 804, respectively, the object identifi- 
cation and region of interest 805 are found from the image at 
time t via feature detection 801 and color segmentation 803. 
The region of interest 805 will be used as sample data to adapt 
new shape and color features for the image at time t+At. 

A simulation using an embodiment of the present invention 
was performed to determine its effectiveness in autonomous 
object recognition. In the simulation, object recognition of a 
moving target was performed. The simulation was based on 
video recorded at a shopping mall in Southern California. 
This application demonstrates that the bio-inspired object 
recognition based on shape and color can be effective to 
recognize an interested person in the heterogeneous environ- 
ment where a single technique, e.g., shape or color, may have 
difficulties in performing effective recognition. This simula- 
tion also demonstrates the mechanism and architecture of an 
autonomous adaptive system having practical uses. 

FIG. 19A shows the initial scene and FIG. 19B shows the 
selected object. When the object is selected, it is extracted to 
obtain its shape and color features. For this simulation, only 
one shape feature and three color features: face skin, hair and 
background, were used. The initial knowledge such as shape, 
face skin color, hair color, and background of the object were 
obtained and used to validate the new scene. FIG. 20 shows 
the sequences of the video. The recognition of the object 
performed by an embodiment of the invention is shown in the 
upper left comer of each image. This sequence of video 
images shows the variation of the dynamics in heterogeneous 
environment and shows that the movements of the person of 
interest to show the rotation and scaling of his face in a 
changing background. Hence, this video represents a realistic 
environment for object recognition. The simulation results 
demonstrate that an embodiment of the present invention 
provides recognition capabilities in a complex situation. 

In a static object recognition situation, a generalized con- 
straint was modified so as to maximize the recognition capa- 
bility of the same object from different perspectives. The 
expectation of having very limited information (one input 
object image) but with a large recognition capability is very 
attractive. Practically and theoretically, a local extrapolation 
of the input object under local rotation and image scaling 
should be expected, but not much more. If so, simulation 
results suggest that embodiments of the present invention 
may be effective as shown in Table I (second row result) with 
no false alarms, full recognition, and a correct locally rotated 
image of the object. This result can be viewed as sampling the 
finite resolution space of a 3-D modeling approach. This 3-D 
modeling can then be used to solve the 3-D rotation of an 
object. 

In pattern recognition, scaling of an object is also an issue 
of great concern and difficult to deal with optimally. From a 
bio -inspired perspective, the bipolar layer can be modeled to 
obtain the scaling ratio between the reference object and the 
potential object. The bio-inspired bipolar layer as an edge 
feature of the object is used in Cellular Neural Network 
(CNN) as described by F. Werblin, T. Roska, and L. O. Chua 
in “The analogic cellular neural network as a Bionic Eye”, 
International J. Circuit Theory and Applications, Vol. 23, pp. 
541-569, 1995. A Visual Cortex-like approach (see, for 
example, Thomas Serre, Lior Wolf, Stanley Bileschi, Maxi- 
milian Riesenhuber, and Tomaso Poggio, “Robust Object 
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Recognition with Cortex-Like Mechanisms”, IEEE Transac- 
tions On Pattern Analysis And Machine Intelligence, Vol. 29, 
No. 3, 41 1 -426, March 2007) has employed multiple window 
resolution and rotation to deal with it effectively; however, 
5 this approach may not be an optimal one. 

Embodiments of the present invention may rely on bio- 
inspired feature extraction based on saccadic eye movements 
and vertebrate retina (horizontal layer), but, generally, no 
scaling or rotations are employed. However, scaling and rota- 
10 tion tolerances are observed in the simulation results shown in 
the figures above. Additional embodiments may tolerate 
increased amounts of rotation. One may effectively employ a 
multiple rotation scheme to cover 3-D rotation. For example, 
if a 30° rotation can be tolerated, it will require 144 (12x12) 
15 rotation positions to cover a complete 360° view of a 3-D 
rotation. In the dynamic object recognition, the local rotation 
and scaling can be tolerated and adaptively learned from the 
adaptive architecture and autonomous feedback mechanism 
according to embodiments of the present invention. 

20 For applications related to dynamic image processing, e.g. , 
object recognition search or visual surveillance and monitor- 
ing, processing time is always a crucial parameter. It is hoped 
that results are obtained within a valid time frame; otherwise, 
they may not be valid. Specially, embodiments of the present 
25 invention do not rely on local spatial -temporal information, 
but, rather rely on global spatial-temporal information of an 
object. Hence, required processing time is more extensive. 
JPL-SPCA demonstrated that about 2-3 min. vs. 6 hours 
(using MATLAB) was needed to extract the features of an 
30 array (100x10000). For the static object search, the time 
consumed is about 0.6 sec to completely examine whether an 
object (e.g., 112x83) is in an image and 3.4 sec of an object 
search (90x60) in an image array for dynamic object recog- 
nition search. These timing numbers above are upper-bound 
35 values and can be reduced by parallel processing, skip frame 
or heuristic reduced search window etc. to meet the timing 
budget for each application. Moreover, hardware approaches 
like 3-D Artificial Neural Network (3-DANN, see T. A. 
Duong, S. Kemeny, T. Daud, A. Thakoor, C. Saunders, and J. 
40 Carson, “Analog 3-D Neuroprocessor for Fast Frame Focal 
Plane Image Processing,” The Industrial Electronics Hand- 
book, Chap. 73, Ed. -In-Chief J. David Irwin, CRC PRESS, 
1997) can improve the speed performance. 

No limitation is intended by the description of exemplary 
45 embodiments which may have included tolerances, feature 
dimensions, specific operating conditions, engineering speci- 
fications, or the like, and which may vary between implemen- 
tations or with changes to the state of the art, and no limitation 
should be implied therefrom. In particular it is to be under- 
50 stood that the disclosures are not limited to particular com- 
positions or biological systems, which can, of course, vary. 
This disclosure has been made with respect to the current state 
of the art, but also contemplates advancements and that adap- 
tations in the future may take into consideration of those 
55 advancements, namely in accordance with the then current 
state of the art. It is intended that the scope of the invention be 
defined by the Claims as written and equivalents as appli- 
cable. It is also to be understood that the terminology used 
herein is for the purpose of describing particular embodi- 
60 ments only, and is not intended to be limiting. Reference to a 
claim element in the singular is not intended to mean “one and 
only one” unless explicitly so stated. As used in this specifi- 
cation and the appended claims, the singular forms “a,” “an,” 
and “the” include plural referents unless the content clearly 
65 dictates otherwise. The term “several” includes two or more 
referents unless the content clearly dictates otherwise. Unless 
defined otherwise, all technical and scientific terms used 
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herein have the same meaning as commonly understood by 
one of ordinary skill in the art to which the disclosure pertains . 

Moreover, no element, component, nor method or process 
step in this disclosure is intended to be dedicated to the public 
regardless of whether the element, component, or step is 
explicitly recited in the Claims. No claim element herein is to 
be construed under the provisions of 35 U.S.C. Sec. 112, sixth 
paragraph, unless the element is expressly recited using the 
phrase “means for ...” and no method or process step herein 
is to be construed under those provisions unless the step, or 
steps, are expressly recited using the phrase “comprising 
step(s) for .... ” 

A number of embodiments of the disclosure have been 
described. Nevertheless, it will be understood that various 
modifications may be made without departing from the spirit 
and scope of the present disclosure. Accordingly, other 
embodiments are within the scope of the following claims. 

What is claimed is: 

1. A method for object identification in multiple images in 
a series of images, the method comprising: 

extracting initial knowledge of color and shape features 
from a statistical image collection and storing the 
extracted color and shape features in one or more data- 
bases; 

performing shape feature detection and color feature detec- 
tion on a first image in the series of images to identify an 
object and a region of interest within the first image, 
wherein the shape feature detection and color feature 
detection is based on the color and shape features stored 
in the one or more databases; 

extracting new color and shape features based on the iden- 
tified region of interest and storing the new extracted 
color and shape features in the one or more databases; 

performing shape feature detection and color feature detec- 
tion on a next image in the series of images to identify the 
object and a region of interest within the next image, 
wherein the shape feature detection and color feature 
detection is based on the color and shape features stored 
in the one or more databases; and 

repeatedly extracting new color and shape features fol- 
lowed by performing shape feature detection and color 
feature identification to identify the object in multiple 
images in the series of images, 

wherein the statistical image collection comprises color 
and shape features mapped into a statistical form based 
on saccadic eye movements. 

2. The method according to claim 1, wherein color feature 
detection comprises color segmentation. 

3. The method according to claim 1, wherein extracting 
shape features and performing shape feature detection com- 
prises performing Principal Component Analysis on the 
region of interest with shape features stored in the one or more 
databases. 

4. The method according to claim 2, wherein extracting 
color features and performing color feature detection com- 
prises processing the region of interest and color features 
stored in the one or more databases with a Cascade Error 
Projection algorithm. 

5. The method according to claim 1, wherein the identified 
object comprises a human being or some portion of a human 
being. 

6. The method according to claim 5, wherein the portion of 
the human being relates to a human head and the new color 
and shape features comprise: head shape; face skin color; hair 
color, and background color. 

7. The method according to claim 5, wherein the portion of 
the human being relates to a human face and the new color and 
shape features comprise: mouth shape and eye shape. 

8. The method according to claim 1, wherein the series of 
images comprises a series of video images. 
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9. A computer programmed for recognition of an object in 
multiple images of a series of images comprising: 

one or more databases for storing color and shape features 
extracted from a statistical image collection containing 
5 images of objects to be recognized; 

one or more processors programmed to extract shape fea- 
tures from a region of interest or an object of interest 
within an image and store the extracted shape features in 
the one or more databases; 

10 the one or more processors being further programmed to 
extract color features from a region of interest or an 
object of interest within an image and store the extracted 
color features in the one or more databases; 

the one or more processors being further programmed to 
15 detect shape features and color features within an image 

or a region of interest based on shape features stored in 
the one or more databases to recognize an obj ect with the 
image or the region of interest; 

the one or more processors programmed to extract shape 
20 features and color features being further programmed to 

extract shape features and color features from the rec- 
ognized object or region of interest, 

wherein the one or more processors are further pro- 
grammed to provide an indication of the presence of the 
recognized object in one or more images of the series of 
images and wherein the statistical image collection com- 
prises color and shape features mapped into a statistical 
form based on saccadic eye movements. 

10. The computer programmed for recognition of an object 
according to claim 9, wherein the one or more processors 

30 further programmed to detect color features are programmed 
to perform detection based on color segmentation. 

11. The computer programmed for recognition of an object 
according to claim 9, wherein the one or more processors 
programmed to extract shape features and further pro- 

35 grammed to detect shape features and color features are pro- 
grammed to perform Principal Component Analysis on the 
region of interest with shape features stored in the one or more 
databases. 

12. The computer programmed for recognition of an object 
40 according to claim 10, wherein the one or more processors 

programmed to extract color features and further pro- 
grammed to detect shape features and color features are pro- 
grammed to process the region of interest and color features 
stored in the one or more databases with a Cascade Error 
45 Projection algorithm. 

13 . The computer programmed for recognition of an obj ect 
according to claim 9, wherein the recognized object com- 
prises a human being or some portion of a human being. 

14. The computer programmed for recognition of an object 
according to claim 13, wherein the some portion of the human 

50 being relates to a human head and the color and shape features 
comprise: head shape; face skin color; hair color, and back- 
ground color. 

15. The computer programmed for recognition of an object 
according to claim 13, wherein the some portion of the human 

55 being relates to a human face and the color and shape features 
comprise: mouth shape and eye shape. 

1 6 . The computer programmed for recognition of an obj ect 
according to claim 9, wherein the series of images comprises 
a series of video images. 

17. The method according to claim 1, wherein performing 
shape feature detection and color feature detection comprises 
selecting multiple samples within the first image or next 
image and wherein locations of the multiple samples are 
selected based on saccadic eye movement. 

18. The computer according to claim 9, the one or more 
65 processors programmed to extract shape features and color 

features being further programmed to select multiple samples 
within the image or the region of interest and wherein loca- 
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tions of the multiple samples are selected based on saccadic 
eye movement. 

19. The method according to claim 17, wherein the statis- 
tical image collection comprises statistical samples of sample 
images and wherein the statistical samples are generated from 
a random pattern based on light intensity. 
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20. The computer according to claim 18, wherein the sta- 
tistical image collection comprises statistical samples of 
sample images and wherein the statistical samples are gener- 
ated from a random pattern based on light intensity. 



